Remote live scene control system, methods, and techniques

ABSTRACT

Apparatuses, methods, systems, and techniques for providing special effects wirelessly using a device plugged into a standard electrical outlet are provided. Example embodiments provide an apparatus and associated software applications for remote and live control of special effects (hereinafter a “Remote Special Effects System,” or “RSES”) using special effect (SE) devices such as individually addressable LEDs, LED strips, fog and smoke machines, and the like. The example RSES described herein comprises one or more SE controller devices that each plug into a standard electrical outlet and are each connected to one or more SE devices. Each SE controller is wirelessly connected to the Internet (or other wide area network) so that it can respond to DMX (or other protocol) commands sent by a remote application by issuing corresponding commands specific to the connected SE device to cause synchronized special effects to occur to an ad-hoc created zone of SE controllers.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Pat. ApplicationNo. 63/248,991, entitled “REMOTE LIVE SCENE CONTROL SYSTEM, METHODS, ANDTECHNIQUES,” filed Sep. 27, 2021, which application is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to methods, techniques, and systemsspecial effects technology and, in particular, to methods, techniques,and systems for location agnostic control of lighting, and other specialeffects such as light wave, electrical, and magnetic device output,haptic feedback, audio, and the like using wireless communication and acontroller device that plugs into an electrical wall outlet.

BACKGROUND

Myriad electronically controlled systems exist for creating specialeffects such as lighting effects for theatre, music and other venues.These systems combine well known protocols for controlling all types oftheatre lighting, fog machines, and other special effects systems. Overthe years, such special effects systems have been wired to be controlledby computing systems that allow a person to control the lights to createeffects (such as flashes, colored lights, gradients, lightning, etc.) orbe programmed to automatically control them based upon timing otherfactors. The DMX protocol was developed initially for theatricallighting and has become a standard for communicating with DMX friendlydevices to cause them to act in certain ways, such as to turn the lightsto a particular color and flash, to turn on a fog machine, or the like.The DMX protocol is to lighting and other DMX special effects devices as“MIDI” is to audio control. The most common lighting control protocolsin use today include Art-Net, sACN/E1.31 and DMX512. Today, most anylighting or other stage effect equipment can be controlled using theseprotocols including moving lights, LED screens, fog and haze machines,and laser displays. Typically the equipment controlled by DMX areconnected together into a universe (of 512 separately addressablechannels) using DMX cables. In some scenarios, DMX consoles have beenreplaced by software, for example running on a personal computer, thatconnects via USB to a control device (such as a DMX USB interface) whichthen communicates through DMX cables to control the lights (or other)devices in that universe. Example DMX Software consoles includeArtNetominator (accessible at“https://www.lightjams.com/artnetominator”), DMXking (accessible at“https://dmxking.com/control-software”), Smart Show (accessible at“http://smartshow.lighting/free-dmx-software/”), and DMX-Workshop(accessible at “https://art-net.org.uk/resources/dmx-workshop/”).

Several DMX over ethernet protocols exist today for communicating tosuch devices over wired Ethernet. Examples of such protocols includeArt-Net, PathPort, ShowNet, sACN, and ETC Net2. These protocols inessence wrap a DMX packet with an IP address to address lightingfixtures such as LEDs in a venue. Art-Net is a UDP based protocol thatgenerally works over a local area network such as an Ethernet. Theyinclude functions such as fader levels for individual lights, positionsof movable lights and management functions for managing nodes in the DMXuniverse. DMX systems such as those described above are often expensive.As well they generally use lighting fixtures that are hard wired toconnect to the DMX controller (DMX console or DMX control device thatconnects to a computer running DMX software console) because such venuestypically require the reliability often associated with wired Ethernetconnections.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof any necessary fee.

FIG. 1 is an example Remote Special Effects System which uses aplurality of special effects controller devices to remotely controlspecial effects devices.

FIG. 2 is an overview flow diagram of how an example Remote SpecialEffects System operates according to an example configuration.

FIG. 3 is a flow diagram of a typical SE-enabled application accordingto an example Remote Special Effects System.

FIGS. 4A-4E are example special effects that can be created by anexample Remote Special Effects System.

FIGS. 5A-5C are additional example special effects that can be createdby an example Remote Special Effects System

FIGS. 6A-6C provide three different views of a printed circuit boardthat implements an Emanator device according to the examples described.

FIGS. 7A-7C provide component layout information for the Emanatordevice.

FIG. 8 is a wiring schematic of an example printed circuit board forimplementing a special effects controller.

FIG. 9 is an example block diagram of an example computing system thatmay be used to practice embodiments of components of a Remote SpecialEffects System.

DETAILED DESCRIPTION

Embodiments described herein provide an apparatus and associatedsoftware applications, methods, and techniques, for remote control ofspecial effects (hereinafter a “Remote Special Effects System,” or“RSES”) using special effect devices such as individually addressableLEDs, LED strips, fog and smoke machines, and the like. Although thisdescription refers in examples to LEDs and other lighting devices, it isto be understood that the other types of devices, such as any IoT (e.g.network addressable) device, can be similarly controlled as long as oneof the special effect devices is individually network addressabledirectly or indirectly and the apparatus has been programmed to addressthe protocol understood by the IoT device. For example, devices such aslight wave (e.g., therapeutic) devices, wearable device outputs (e.g.,haptic feedback, LEDs, audio), and the like can also be controlledthrough a Remote Special Effects System using the techniques describedhere regardless of the type of network they are on (e.g., wirelesstraditional or mesh or wired).

The example Remote Special Effects System described herein comprises oneor more special effect controller devices (SE controllers) that eachplug into a standard (residential or business) electrical outlet and areeach connected to one or more special effects devices (SE devices) suchas an LED strip, a fog machine, a horn, a wavelength output device, andthe like. Each SE controller is wirelessly connected to a network suchas the Internet (or other local area, wide area, or other network),directly or indirectly (e.g., through another SE controller), so that itcan respond to DMX (or other protocol) commands sent by an applicationremote to the SE controller by issuing corresponding commands specificto the connected SE device to cause a special effect to occur. Thus,existing AV tools used by AV designers to emit DMX commands can beeasily integrated into the RSES to control special effects for use bythe general public (household or business or any use) -no specializedknowledge of the SE controller is required to generate these specialeffects. For example, if the special effects device is an LED strip ofNeopixel LEDs produced by Adafruit Industries (which control RGB or RGBWLEDs using a single wire), then the SE controller can receive a packetwith DMX commands (for example using an Art-net protocol packet whetherit is from an existing AV tool or a new application), and thenautomatically issue corresponding driver instructions (e.g. WS28xx,WS2812, or other driver instructions) to address one or more of theNeopixel LEDs.

In an example embodiment, the SE controllers are printed circuit boards(“PCBs”) that can be encased in available housings similar to the sizeof an AC adapter. Because these SE controllers are small devices thatcan utilize standard electrical outlets for power, they can be producedinexpensively thereby enabling theatrical quality lighting and otherspecial effects to be produced at home and for the masses. Moreover,because they connect wirelessly to a network such as the Internet, theycan be controlled remotely by applications, similar to most IoT devices(they can behave and be controlled as an IoT device). In addition,because of their wireless connectivity, the special effects devices canbe placed in locations devoid of Ethernet, including locations that arecontrolled by different types of wireless networks such as traditional(centralized) networks and decentralized (mesh) networks. As well, thespecial effects devices controlled need not be DMX enabled or connectedby unwieldly Ethernet cables in a semi-permanent (fixed for a time)configuration such as commonly found in a theatrical or otherprofessional setting.

In other example embodiments, the PCB (SE controller) may be powered bya battery contained within, adjacent, or proximate to a housing. Thisenables some example SE controllers to be embedded in devices (ratherthan ‘plugged in’ to a wall socket) which communicate wireless, forexample, with a remote application that controls special effect devicesmanaged by the PCB controller. The SE devices may be resident on thesame device that houses the PCB controller.

Using a wireless solution rather than hard-wired or Bluetooth solutionseliminates a need to have devices controlled together within a certainrange, such as supported by Ethernet cables or Bluetooth’s distancerequirements. Rather, in some scenarios, the SE controllers can bejoined together in an “ad-hoc” (at the time or as-available) fashion andsynchronized (by another program) to create joint special effects indiscrete and disjoint (not connected) physical locations, in the samephysical location, in a virtual location, and/or across one or moredevices. For example, in a virtual classroom where each student has atleast one SE controller connected to a special effects device (such as alight strip or audio producing device), synchronized special effects canbe triggered by the teacher through software programmed to communicatewith the joined SE controllers, for example, to make sure the studentsare awake and paying attention. As another example, a virtualpresentation with slides (e.g., delivered via a video conference) can beprogrammed (pre-programmed or controlled on the fly) to cause specialeffects as individualized or synchronized animations that occur in eachphysical or virtual location from where each viewer is participating(e.g., room, office, home, etc.). The special effects may even betargeted to an individual viewer. Or, they may be employed in a crowdsituation such as to cause participants to group together by producingunique special effects to the target participants for each intendedgroup. In addition, in some scenarios, the participants may have jointcontrol over the special effects.

In other examples, sensor input (such as a weather station) may beobtained from an application, which generates special effects responsiveto detection of certain conditions. For example, sensors connected to ahuman body that measure attributes such as heartrate, perspiration,blood flow, and the like may be connected to applications that use an SEcontroller plug-in to communicate with SE controllers connected todifferent special effects (output) devices. Other sorts of sensor input,for example input from wave length devices, magnetical, electrical,optical, and other devices, can be similarly accommodated. Accordingly,in some scenarios SE devices can be controlled and operate withoutdirect human manipulation but rather from sensing a condition from asensor that measures some characteristic of somebody or the environment.

In yet other scenarios, different special effect devices may besynchronized and coordinated such as might be useful in a teaching orpresentation. For example, audio special effect devices may besynchronized to lighting or haptic feedback special effect devices asdesired by the application controlling the SE controllers attached tothese devices. In another example, proximity sensors may be located ondevices or people or at locations that cause input to an SE controllerenabled application. Upon sensing that two people/devices are inproximity of one another (or in proximity to some location), theapplication sends commands to one or more SE controllers which in turncontrol one or more (output) SE devices to produce synchronized orcoordinated special effects.

Accordingly, there are many possible work related, home, and/orentertainment related uses of such SE controllers and related software.

In addition to these advantages over existing systems, other advantagesare presented because the SE controllers are connected wirelessly to theInternet (or other local or wide area network) directly or indirectly(e.g., through another SE controller) and they accept any type of liveDMX packet stream over a wireless network (e.g., using Art-Net or otherprotocol). For example, animations are not limited to pre-programmedlighting displays - a new and different experience can be offered eachtime an event is run. In addition, animations, music, or sound can besourced live from anywhere in the world as long as the input source isInternet accessible. Further, the SE controllers can run any ofapproximately 2.8 million color combinations - limited only by theintelligent lighting apparatuses communicatively connected to the SEcontrollers.

Each SE controller is wirelessly connected to the Internet (or otherwide area network or to a local network) so that it can respond to DMX(or other protocol) commands sent by an application remote to the SEcontroller by issuing corresponding commands specific to the connectedSE device to cause a special effect to occur. The wireless connectionmay be a traditional (centralized) wireless network or a decentralized(e.g., mesh) wireless network. In addition, although example embodimentsof the RSES are described for controlling SE devices that use DMXprotocol (by wrapping them in Art-net packages for transport), it is tobe understood that they system architecture and ideas presented herecould be used to extend control to other types of devices controlled byother than DMX protocols. Similarly, other packet wrapper protocolsbeyond Art-net could be incorporated into an RSES and provide thefunctional benefits described herein.

FIG. 1 is an example Remote Special Effects System which uses aplurality of special effects controller devices to remotely controlspecial effects devices, for example in a live manner targeted todiscrete locations. As shown in FIG. 1 , RSES 100 comprises one or morespecial effects controller devices 120-123 (SE controllers) connectedwirelessly to wide area network 110, typically the Internet, andcommunicatively connected to one or more respective special effectsdevices 106-109. Each SE controller 120-123 is also connected to acorresponding electrical outlet 105 a-105 d for power. The SEcontrollers may reside in locations potentially geographically remotefrom each other (such as located in different physical residences,businesses, regions, or countries, e.g., having different postaladdresses). For example SE controller 120 connected to LED strip 106 maybe installed in student A’s residence whereas SE controller 121connected to LED strip 107 may be installed in student B’s residence.Furthermore, other SE controllers, such as SE controller 122, may beinstalled in a different state or potentially a different country, in abuilding or other structure, or location not associated with a buildingprovided that WiFi is available. Also, the SE controllers may share thesame IP address or may have distinct IP addresses. The SE controllers120-123 may be connected to a variety of special effects devicesincluding, for example, individually addressable (RGB) LED strips 106and 107, Leko light 108, or standard (dumb) RGB LED strip 109. Otherdevices such as fog and smoke machines, audio output devices, hapticfeedback devices, other mechanical, electrical, and/or light wavedevices, and myriad sensors and devices can be similarly connected andare not shown.

The SE controllers 120-123 are connected wirelessly to a special effectsanalyzer and control program (SE analyzer) 101 or an other specialeffects-enabled program 130 via network 110. These programs may executeon any type of computing device. The SE analyzer 101 demonstrated here,which in an example embodiment described below is known as the RemoteLive Scene Control or “RLSC” application, is configured to analyze soundand send control data (such as lighting control commands) to each of theSE controllers 120-123 individually and can synchronize the behavior ofthese SE controllers 120-123 such that all or some of the connectedspecial effect devices 106-109 (e.g., lights) react/behave in the sameor different manner at the same or different times. In this way the SEcontrollers 120-123 can cause special effects that are synchronized orotherwise timed to cause a similar behavior to an audience even thoughthe audience members are not present in the same physical venue and theSE controllers are thus in remote locations from each other. Of coursethe special effects can also synchronize special effects to one or morespecial effects devices in the same physical venue. Here venue refers toa physical address such as associated with a building, residence, event,open space, field, etc. Thus, the Remote Special Effects System can beused to unite an audience (through special effects) whose members areconnected via a virtual venue such as a video conference over web basedvideo conferencing software, or a presentation to a virtual audienceconnected to the venue through their computers. In addition, the RSEScan also be used in a standalone environment such as to createtheatrical and event quality special effects in one’s one residence orbusiness location.

As well, SE controllers 120-123 may respond to commands from a singlepresenter (such as a teacher or speaker), several presenters (e.g., aband), and/or one or more participants, for example in an interactiveclassroom, a family reunion, or a crowd. In addition, any of the effectsstreamed to the SE controllers 120-123 may be predesigned and thus“played back” or may be generated on the fly. All such combinations arecontemplated.

In one example embodiment, the SE analyzer 101 is used to analyze sound,such as a song, soundtrack from a movie, a presentation, or any audiotrack and to cause special lighting effects such as using LED strips tochange colors in a sequence or randomly or to produce some other type oflighting animation in conjunction with attributes of the sound, such astone/pitch (measured for example as frequency) and loudness (measuredfor example as decibels). For example, based upon detection of aparticular frequency, a corresponding color command(s) may be sent to anindividually addressable LED, group of LEDs, or the entire strip. In theexample system shown in FIG. 1 , the SE analyzer 101 has beenspecifically programmed to generate commands via Art-Net (DMX overethernet), which commands are sent to the various independentlyaddressable SE controllers 120-123. Other commands according to otherprotocols (such as sACN, a streaming protocol that uses a multicastcommunication technique) may be similarly incorporated. Each SEcontroller 120-123, when it wirelessly receives an Art-Net (or otherprotocol) packet, translates the packet to an appropriate deviceprotocol understood by the special effects device communicatively (insome cases electronically) coupled to the SE controller. For example inthe case of an RGB pixel addressable LED strip 109 (such as Neopixel LEDdevices by Adafruit Industries), the SE controller 123 receives Art-Netpackets and translates DMX lighting commands to a protocol understood bythe ws2812 driver, used to control the Neopixel device 109.

Other third party programs and applications such as third party program130 can used RSES application programming interfaces (APIs) andlibraries to produce Art-Net (or other protocol) packets to control theRSES SE controllers 120-123. Thus, a mobile or other computer basedapplication can be used to create readily accessible special effects,for example to play music with live special lighting effects that arecreated dynamically (on-the-fly) in response to analyzing the audio.Further, the same audio may be analyzed differently each time the musicis played allowing different special effects experiences each time.Alternatively, predesigned lighting effects, or A/V artist designedlighting effects may be played alongside the music as it is played. Insome scenarios, the SE analyzer and/or 3^(rd) party program 130 mayincorporate data or stored configurations from data repositories 102.

As mentioned, the SE controllers 120-123 are location agnostic and canjoin a special effects “session” on an “ad-hoc” basis. For the purposeof this description, a session is defined as those SE controllers thatare accessible and addressable to an SE analyzer 101 or specialeffects-enabled application 130 at a particular time. The SE controllers120-123 can be unified by a special effects service (SE service) 103,for example a web service in a more centralized computer architecturescheme, that controls a concept of a session - which are the currentlyaddressable SE controllers that the SE analyzer can control. Since theSE controllers may be plugged in and disconnected on the fly, the SEservice maintains a concept of a current “session” for the SE analyzer101. Alternatively, any special effects-enabled application, such as theSE analyzer 101 or the 3^(rd) party program 130 may integrate their own(wireless) discovery and registration process such as using a mixture oflogin capabilities, scan and handshake protocols, etc. Any type ofdiscovery and registration process may be used by the SE service 103 orby these programs to discover available SE controllers and/or toregister them as part of a current session. In addition, any specialeffects-enabled program (such as SE analyzer 101 or program 130) canpresent a user interface for configuring users, SE controllers,presenters, etc. Thus, the SE analyzer 101 or special effects-enabledprogram 130 can not only cause special effects to happen dynamically(e.g., while analyzing an audio stream) but it is also “ad-hoc” andcontrols whichever SE controllers 101 are currently connected to theapplicable session managed by the discovery and registration process. Asan alternative to using a web service, the RSES arrangement can beadapted to a peer-to-peer system, where one of the SE controllers120-123 becomes the server (or a “master”) and acts to control thesession aspect. In addition, the SE controllers that comprise the RSESarrangement can operate as a mesh network where each SE controller actsas a node, for example, using a protocol and API such as ESP-WIFI-MESH.Other suitable arrangements are contemplated, such as those that arestrictly used in an Intranet scenerio.

FIG. 2 is an overview flow diagram of how an example Remote SpecialEffects System operates according to an example configuration, such asthat shown in FIG. 1 , in order to generate synchronized special effectsto an ad-hoc “session” (which can also form an ad-hoc network orsubnetwork) of special effects devices. Here “synchronized” refers totimed special effects streams (which are coordinated according to aparticular time, in an order or sequence, relative to each other, andother variations) which are forwarded to one or more SE controllers tocoordinate special effects in potentially different physical (postaladdress or other lat/long designation) locations. The SE controllersalso may reside in the same physical location, at the same IP address,or at different IP addresses, and any other such combination. Ad-hocrefers to a concept in which a “zone” of special effects devices(through their communicatively connected SE controllers) is created andmade known to the SE analyzer 101 or 3^(rd) party program 130, to whichthe special effects instructions are forwarded (e.g., sent, transmitted,multicast, broadcast, etc. depending upon the application and protocolbeing used). In block 201, an SE-enabled application such as SE analyzer101 or 3^(rd) party program 130 runs an application where a specialeffect is selected (e.g., a color scroll, or a “plasma” effect or aselected color) either programmatically or potentially by a user using auser interface or other tool. In response, the SE-enabled applicationsends out a DMX packet via WiFi (block 202) through Art-Net (or otherprotocols) to the network addresses (e.g., IP addresses, MAC addresses,etc.) associated with the SE controllers that have been registered witha current session (e.g., by SE service 103 or by an SE-enabledapplication with such capability). In blocks 203 and 204, each SEcontroller that is part of this ad-hoc session receives the packet andtranslates it into data that the communicatively connected SE device(e.g., LED strip 106) understands. The corresponding SE devices (blocks205 and 206) then receive this packet (typically via a software orfirmware driver associated with the respective device) and performs thespecial effects (e.g., lighting) command issued by the corresponding SEcontroller. Other types of SE devices and other types of special effectscan be handled similarly.

FIG. 3 is a flow diagram of a typical SE-enabled application accordingto an example Remote Special Effects System. The SE-enabled application300 depicted is for example the sound analyzer effects control program101 (called a Remote Live Scene Control application below) or a thirdparty RLSC-enabled application such as SE-enabled application 130. Here,in logic block 301, the application discovers (e.g., via a web service103) or otherwise obtains a list of currently connected SE controllersto which special effects are to be transmitted. This list forms an“ad-hoc” network (e.g., a synchronized special effects zone) ofcontrollers for synchronized special effects. In block 302, theapplication receives an indication from a user (or a program such as aslide presentation program) of a special effect type and attributes.Other characteristics may also be included such as timing specifics,target SE controllers, and the like. An example Application ProgrammingInterface (API) is described below for use by an SE-enabled application.In block 303, the application generates an Art-Net (or other protocol)packet that includes a computer understandable special effect (e.g.,using DMX or other protocol). In block 304, this special effect packetis transmitted to some or all of the SE controllers in the ad-hocnetwork.

These packets are then received and processed by the SE controllers asdescribed according to FIGS. 1 and 2 .

FIGS. 4A-4E are example special effects that can be created by anexample Remote Special Effects System as described herein. Here, eachfigure is a snapshot in a time series of a changing color animationproduced by an SE controller using an installation of LEDs surrounding aperson’s computer screen. Using the RSES, in an environment that hasmultiple participants and different physical locations, each having acomputer screen arranged similarly, an application can be used totrigger this special effect across the entirety of the ad-hoc connectedSE controllers thus having participants have the same special effectsproduced around their computer screen at the same time or in a timedsequence (for examples to create a “wave” effect), or the like. Or theapplication can select and produce different special effects tailored toone or more of the participants and coordinate them in any manner (forexample, to give focus to a particular participant or other speaker).For this reason, the example implementation of an SE analyzer isdescribed below as “remote live scene control.”

FIGS. 5A-5C are additional example special effects that can be createdby an example Remote Special Effects System as described herein. Here,each figure is a snapshot in time of color rolling through an LED striplight attached to an SE controller. This special effect can becoordinated across multiple participants connected virtually asdescribed according to FIGS. 4A-4E.

Of note, any of these special effects, applications, SE devices and SEcontrollers are also operable within a single physical address - forexample, one or more SE controllers in a single (physical or virtual)room, multiple SE controllers connected to different devices in a singlehousehold or business, and any other combination. In addition, as newspecial effects devices that understand protocols such as DMX or otherprotocols become available, the SE controllers can be adapted (e.g.,through updates to firmware) to translate DMX commands to a language(commands) that the drivers for the devices understand. Thus, the SEcontrollers are not limited to receiving DMX packets and generating anyparticular lighting commands (such as ws2812). Accordingly as newspecial effects are created for example using sensors or otherapparatuses, the SE controllers can accommodate them to produce other orimproved special effects.

As well, a current embodiment of the SE analyzer (the RLSC analyzer)produces DMX packets and transmits them using Art-Net protocol (IPaddressable DMX). This sound analyzer and other applications canincorporate other protocols as they are developed and the SE controllerfirmware updated. Accordingly, the RSES described here may be expandedto incorporate different and other protocols and other devices.

Example embodiments described herein provide applications, tools, datastructures and other support to implement a Remote Special EffectsSystem to be used for live or pre-programming special effects to bedelivered wirelessly in a synchronized fashion to specific devices inseparate physical locations. Other embodiments of the describedtechniques may be used for other purposes. In this description, numerousspecific details are set forth, such as data formats and code sequences,etc., in order to provide a thorough understanding of the describedtechniques. The embodiments described also can be practiced without someof the specific details described herein, or with other specificdetails, such as changes with respect to the ordering of the logic,different logic, etc. Thus, the scope of the techniques and/or functionsdescribed are not limited by the particular order, selection, ordecomposition of aspects described with reference to any particularroutine, module, component, and the like.

Example RSES Embodiment

An example RSES embodiment with the components described according toFIGS. 1-3 is directed to producing special effects using lighting andthe DMX protocol (via Art-Net protocol). This example embodimentincludes one or more SE controllers, which are referred to as Emanatordevices. These are the devices that plug into a standard electricaloutlet and control a lighting device such as an LED strip. In operation,a participant who is participating in a session managed by the RLSCapplication plugs the special effects devices (such as an intelligentlight or fixture) into an Emanator device, and then plugs the Emanatordevice into a wall electrical outlet.

The example RSES embodiment also includes an application, the RemoteLive Scene Control application (or RLSC application), which isconfigured to produce special effects as described according to FIGS. 2and 3 . The example RLSC produces Art-Net packets with DMX commands,however can be modified as described above. In addition, the exampleRSES embodiment includes an API which can be used to develop otherspecial effects-enabled applications.

Other example components and embodiments are of course possible to bedeveloped to produce synchronized lighting effects and other specialeffects as described herein.

Remote Live Scene Control Application

The example RLSC application, which may be provided via a mobile, IoT,or wired computing system, is designed to configure and organize one ormore Emanator devices and stream effects to each Emanator device orcombine them all into one or multiple synchronized special effectszones. The application includes user and device discovery features foreasy setup and facilitates organizing and combining devices into zonesfor further custom control. The app provides various pre-programmedeffects including some that will react to live audio and producelighting (or other scene) effects synchronized to audio. The RLSC usermay also configure control of the various participants’ special effectsdevices over the Internet for applications such as live virtual meetingsor concerts. In some examples, the RLSC application can be operated toproduce on-the-fly (dynamic) special effects as well as pre-programmed,pre-designed, or pre-recorded effects. In some scenarios, thesepre-programmed/designed/recorded special effects may be obtained from AVdesigners who have produced them for theatre or movies using industrystandard DMX consoles or equivalents.

In one embodiment the RLSC application includes (or is accompanied by aseparate) RLSC Music Analyzer which analyzes sound based upon frequency(or pitch) and loudness as described with reference to FIG. 1 . The RLSCMusic Analyzer is an example of a third party application that can bewritten once it is special effect-enabled - that is using the Emanatordevice API. Using the Emanator API, the third party application canstream data in real time and send it to one or more Emanator devices.

As described, an application such as the RLSC application can be used togenerate and control special effects for a synchronized special effectszone of a plurality of Emanator devices (which may reduce to one specialeffects device connect to one plugged-in Emanator). This capability isreferred to as “remote live scene control.” For example, an RLSCapplication may:

-   Run an application with remote live scene control;-   Play a movie with remote live scene control;-   Play music with remote live scene control;-   Conduct a virtual meeting with remote live scene control; or-   Facilitate a group gathering with remote live scene control.

Many examples include the concept of no one missing the event becausethey cannot attend physically - this is referred to as “virtual liveimmersion.” The following scenarios are just some of these contemplatedexamples in addition to those summarized above:

-   Play music to artist supplied lighting effects:    -   Play your music with live analyzed lighting effects. Different        analyzed lighting effects can offer different experiences each        time music is played; or    -   Play your music with live pre-designed light effects, e.g., A/V        artist designed lighting effects. (MP3 & DMX lighting effects        together)-   Virtual meetings with “at home” lighting synchronized with    presentation. Slideshow initiated preprogrammed effects.    -   Grab your virtual audience’s attention with a stroke of        lightening & thunder effects happening in the room - not just        the screen.    -   Your sound, prerecorded voice or video, with live analyzed        effects. Duplicated playbacks can vary based on sound analyzer        or pre-designed Light effects.-   Other Virtual Examples include:    -   Waiting on hold for any service (e.g., medical)    -   Soothe online patients with music set to reduce stress        scene/light displays.    -   Virtual school with teacher presentation synchronized in        students’ room at home, dorm, or any location, examples:        -   Synchronize attendee lighting with the push of a button to            wake up snoozing students during class with synchronized            lighting across all attendees. Picture effect on all            attendees screens, the all the virtual attendees’ squares            can be sync’d with the lighting in the room.        -   Flash lights on one attendee to gain attention or focus            attention.    -   Virtual Family Reunions        -   Play family games (Trivia, Jeopardy) with live player(mic)            sound, music, or noise analyzed live with lighting            responding across all family’s homes.    -   Virtual Church or School Choirs        -   Virtual choir analyzed with live lighting responding on all            device screens as well as in the home.        -   Choir with live pre-designed light effects. e.g., A/V artist            designed lighting effects. (MP3 & DMX lighting effects            together)    -   Virtual DJs        -   Virtual dances music & sound analyzed with live lighting            responding on all screens as well as in each home or            location.        -   Virtual weddings with preprogrammed lighting effects            occurring in each location. No one misses the event; they            were immersed in it virtually. All participants can effect            and create the event.    -   Virtual Theatre        -   Virtual Play with professional dramatic stage effects            occurring in location home, rented space, etc.        -   All virtual participants can effect and create the event.    -   Virtual Theatre with pre-designed light effects. e.g., A/V        artist designed lighting effects. (MP3 & DMX lighting effects        together)-   Movies come alive in your home -    -   Experience your home lighting or devices responding to the crack        of sound or lightening during a Harry Potter movie or blinding        light for hyperdrive in Star Wars. Theatrical special effects        brought to the masses.    -   Movies download with pre-designed light effects. e.g., A/V        artist designed lighting effects. (Video file format & DMX        lighting effects together)-   Virtual Concerts    -   Attend concerts with stage effects happening in your location,        home, backyard, any location. Music analyzed on the fly or        pre-designed light effects. e.g., A/V artist designed lighting        effects.-   Gaming    -   Not just stagnant LEDs behind your monitor but light that reacts        to game events.    -   Game music analyzed lighting effects.    -   pre-designed light effects. e.g., A/V artist designed lighting        effects integration into the game-   Presentations    -   Videos or music included in presentation synchronized with room        lighting    -   Analyzed effects for music or noise from presenter (pragmatic        pauses) or audience (applause)    -   Pre-designed effects to wake-up audience    -   Sensors hooked-   Key activated lighting in applications    -   E.g., Running/Exercise        -   The RSES technology can be used to create cost effective            personal pace trainers for runners or cheaper alternatives            for schools or lower budget training operations.        -   Pacing machine effects using portable waterproof LEDs        -   Replace existing and expensive light training products with            a more cost effective solution (costing tens of thousands of            dollars for equipment & installation costs) for improving a            runner’s pace. Example, sections of lights can be set up            around a field or track and a smartphone or computer can            synchronize a pace light, section of illuminated LEDs, where            runners must try and run as close to the lights as possible.-   Grid LEDs    -   Times Square like effects for advertising and other purposes

Emanator (Special Effects Controller)

An example RSES embodiment provides an Emanator device as the SEcontroller. FIGS. 6A-6C provide three different views of a printedcircuit board that implements an Emanator device according to theexamples described. The device includes a WiFi chip, a microprocessor, afan, an AP (access point) mode header and various buttons and othercomponents including firmware. FIGS. 7A-7C provide component layoutinformation for the Emanator device. Other embodiments that are poweredfrom proximate battery power can be similarly accomodated.

In summary of operation, 12 V power enters the PCB and goes through somepower conditioning to prevent spikes and elongate the life of the LEDs.In its current configuration, the board contains a Wemos D1 Mini whichincludes an ESP8266 WiFi chip for wireless connectivity. In otherconfigurations (not shown), the board is configured to include an ESP32chip which is more powerful, dual-core, and provides other functionalitysuch as Bluetooth for configuration purposes. Other chips can besimilarly incorporated. The PCB receives Art-Net data packets throughWiFi and the firmware on the ESP8266 chip processes it and outputs datathe LEDs can understand (e.g., WS28xx driver commands for NeopixelLEDs). There is a terminal block header which delivers power and data tothe LEDs. Additionally, the board contains fan, reset, and AP-modeheaders. The fan header provides configurable 5 or 12 volts for anoptional cooling fan. The reset header provides a connection for anexternal reset button, which resets ESP8266 chip. The AP-mode buttonprovides a connection for an external button that can force the boardinto configuration mode to reconfigure the wireless connection.

The Emanator firmware on the PCB has an Art-Net receiving library (e.g.,from public domain software) which takes the WiFi delivered Art-Net dataand provides the raw DMX, which is sent to the light or fixture todisplay requested color/state.

In an example Emanator device that is connected to NeoPixel LEDs, thenthe Neopixel library (e.g., from public domain software) resident on thePCB generates the ws28xx compatible data (DMX data sends digital signalsover the pin out in the order it needs to light the lights as requested(0-N# lights out of one pin) - telling them one of the millions of RGBvalues).

When the PCB receives a Wi-Fi transmission, the firmware processes thepacket using the Art-Net receiving library and sends the signals to thelights, for example using the Neopixel library. In this manner, theEmanator PCB acts as a translator from DMX commands to lighting devicecompatible commands. Thus, the Emanator firmware turns (e.g., aNeopixel) LED device into a network addressable DMX device. Othertranslations can be programmed into the Emanator PCB.

In its current implementation, the PCB firmware is configurable using aweb-based user interface. Config settings include what number of LEDSare connected to the board, which pin is connect to the LEDs, 2 statusindicators that can be turned off, WIFI network connected, IPaddress,Art-Net universe information, and the like. An RLSC application or otherSE-enabled application may be programmed to configure Emanator devices.

FIG. 8 is a wiring schematic of an example printed circuit board forimplementing a special effects controller. In summary, power comes inand is funneled thru a capacitor which cleans the power. It then goesthru a fuse to prevent drawing too much power and damaging thecontroller. Then 12 v power goes to LEDs. In one embodiment, the PCB hasa fan header to power a 5 v or 12 v fan and a reset button header. Italso has an AP mode header to turn the PCB into Access Point mode toreconnect WiFi. The PCB also includes a 12 v to 5 v stepdown, to steppower for WEMOS board which has the ESP8266 chip on it. Othermicrocontrollers and other WiFi chips can be similarly supported. Thelogic shifter is used to shift 3.3 volt out of WEMOS to 12 volt logicthe LED strip needs. The PCB includes LED indicator stack lights with 2resistors that act as status indicators for the board. There is alsoanother resistor to clean data signal from WEMOS to the LEDS. The datasignal to the lights, of 24 × n bits, (where “n” is the number of lightsin the LED strip and 24 bit provide 24-bit color to a single LED) is runthru a resistor to purify the signal and reduce noise.

The bulk of the PCB is power delivery to lights and electrical safetyfeatures. Specifically, the PCB moderates the lights from pulling toomuch power from the power supply, thereby reducing LED burnout and powersupply overheating which can cause a potential fire. The PCB contains acapacitor to prevent spikes in power to the LEDs and a fuse to preventburning out the power supply.

The PCB can be programmed to control any IoT (internet of things)device, any network addressable device, including another computingdevice.

RSES Application Programming Interface (API)

An example RSES embodiment also includes an application programminginterface defining functions for facilitating the development andpublication of special effect-enabled applications.

Table 1 below includes an example set of interfaces. It is to beunderstood that variations are possible and can be incorporated. Inaddition, functions may be added, modified, or removed.

TABLE 1 Resource (piece of data) POST GET User Interface PUT DELETE/userid -possibly login login Create a new user with Experiential Modeof Userid RLSCInputActive Y/N Define Userid Bulk update of userid Removeall userid’s Sole Presenter JointPresenter Crowd/Class PresenterAudience Member (default) /stream Define Audio/Video Stream PreDesigned,Adhoc, Combo Define Stream Bulk Update Stream w PreDesigned SE Stream wAdHoc SE (default) Combined Stream in Zone Zone Settings /RSLC DeviceTypes Define Devices Available for Analysis Define Device Update detailsof userid LED Lights (default) Fog Machines Motion Sensor TemperatureSensor /RSLC Controller Adjustment Control for RSLC Devices AdjustDevice Update On/Off Settings Wifi Brightness Loudness Moodness etc/RSLC Analyzer Adjustment RSLC Control for Analysis Adjust Device On/OffSettings Brightness Loudness Mood Options etc /Stream State Streamingwith End User Control Stream Control Play (default after timeout) PauseFF RW /Shared Devices Devices Across Zone Avail with for each SettingDefine Device LED Lights (default) DMX Device Settings Sensor Owner TypeProtocol Shared etc

Example Computing System

FIG. 9 is an example block diagram of an example computing system thatmay be used to practice embodiments of components of a Remote SpecialEffects System described herein, such as the Special Effects Analyzerand Controller (SE Analyzer 101 of FIG. 1 ), or other third partyspecial effects-enabled applications (application 130 of FIG. 1 ). Notethat one or more general purpose virtual or physical computing systemssuitably instructed or a special purpose computing system may be used toimplement an RSES. Further, components of the RSES may be implemented insoftware, hardware, firmware, or in some combination to achieve thecapabilities described herein.

Note that one or more general purpose or special purpose computingsystems/devices may be used to implement the described techniques.However, just because it is possible to implement the SE analyzer on ageneral purpose computing system does not mean that the techniquesthemselves or the operations required to implement the techniques areconventional or well known.

The computing system 900 may comprise one or more server and/or clientcomputing systems and may span distributed locations. In addition, eachblock shown may represent one or more such blocks as appropriate to aspecific embodiment or may be combined with other blocks. Moreover, thevarious blocks of the SE analyzer/RLSC application 910 may physicallyreside on one or more machines, which use standard (e.g., TCP/IP,UDP/IP) or proprietary interprocess communication mechanisms tocommunicate with each other.

In the embodiment shown, computer system 900 comprises a computer memory(“memory”) 901, a display 902, one or more Central Processing Units(“CPU”) 903, Input/Output devices 904 (e.g., keyboard, mouse, CRT or LCDdisplay, etc.), other computer-readable media 905, and one or morenetwork connections 906. The SE analyzer/RLSC application 910 is shownresiding in memory 901. In other embodiments, some portion of thecontents, some of, or all of the components of the SE analyzer/RLSCapplication 910 may be stored on and/or transmitted over the othercomputer-readable media 905. The components of the SE analyzer/RLSCapplication 910 preferably execute on one or more CPUs 903 and managethe generation of special effects in a synchronized zone across awireless network, as described herein. Other code or programs 930, theRSES web service 940, RSES API definitions and libraries 917, andpotentially other data repositories, such as data repository 920, alsoreside in the memory 901, and preferably execute on one or more CPUs 903as required. Of note, one or more of the components in FIG. 9 may not bepresent in any specific implementation. For example, some embodimentsembedded in other software may not provide means for user input ordisplay.

In a typical embodiment, the SE analyzer/RLSC application 910 includesone or more sound analysis engines 911, one or more DMX or otherprotocol generation engines (or libraries) 912, a data repository 915 ofspecial effect information, scripts, configuration parameters, etc., andother components such as a different special effects engine 913. In atleast some embodiments, the other special effects engine 913 is providedexternal to the SE analyzer/RLSC application and is available,potentially, over one or more networks 950. Other and /or differentmodules may be implemented. In addition, the SE analyzer/RLSCapplication 910 may interact via a network 950 with other specialeffects-enabled applications or client code 955, one or more SEcontrollers (Emanators) 960, and/or one or more third-party informationprovider systems 965, such as pre-designed AV effects. Also, of note,the 915 data repository may be provided external to the SE analyzer/RLSCapplication as well, for example in a knowledge base accessible over oneor more networks 950.

In an example embodiment, components/modules of the SE analyzer/RLSCapplication 910 are implemented using standard programming techniques.For example, the SE analyzer/RLSC application 910 may be implemented asa “native” executable running on the CPU 103, along with one or morestatic or dynamic libraries. In other embodiments, the SE analyzer/RLSCapplication 910 may be implemented as instructions processed by avirtual machine. A range of programming languages known in the art maybe employed for implementing such example embodiments, includingrepresentative implementations of various programming languageparadigms, including but not limited to, object-oriented, functional,procedural, scripting, and declarative.

The embodiments described above may also use well-known or proprietary,synchronous or asynchronous client-server computing techniques. Also,the various components may be implemented using more monolithicprogramming techniques, for example, as an executable running on asingle CPU computer system, or alternatively decomposed using a varietyof structuring techniques known in the art, including but not limitedto, multiprogramming, multithreading, client-server, or peer-to-peer,running on one or more computer systems each having one or more CPUs.Some embodiments may execute concurrently and asynchronously andcommunicate using message passing techniques. Equivalent synchronousembodiments are also supported.

In addition, programming interfaces to the data stored as part of the SEanalyzer/RLSC application 910 (e.g., in the data 915) can be availableby standard mechanisms such as through C, C++, C#, and Java APIs;libraries for accessing files, databases, or other data repositories;through scripting languages such as XML; or through Web servers, FTPservers, or other types of servers providing access to stored data. Thedata repository 915 may be implemented as one or more database systems,file systems, or any other technique for storing such information, orany combination of the above, including implementations usingdistributed computing techniques.

Also the example SE analyzer/RLSC application 910 may be implemented ina distributed environment comprising multiple, even heterogeneous,computer systems and networks. Different configurations and locations ofprograms and data are contemplated for use with techniques of describedherein. In addition, the [server and/or client] may be physical orvirtual computing systems and may reside on the same physical system.Also, one or more of the modules may themselves be distributed, pooledor otherwise grouped, such as for load balancing, reliability orsecurity reasons. A variety of distributed computing techniques areappropriate for implementing the components of the illustratedembodiments in a distributed manner including but not limited to TCP/IPsockets, RPC, RMI, HTTP, Web Services (XML-RPC, JAX-RPC, SOAP, etc.) andthe like. Other variations are possible. Also, other functionality couldbe provided by each component/module, or existing functionality could bedistributed amongst the components/modules in different ways, yet stillachieve the functions of an SE analyzer/RLSC application.

Furthermore, in some embodiments, some or all of the components of theSE analyzer/RLSC application 910 may be implemented or provided in othermanners, such as at least partially in firmware and/or hardware,including, but not limited to one or more application-specificintegrated circuits (ASICs), standard integrated circuits, controllersexecuting appropriate instructions, and including microcontrollersand/or embedded controllers, field-programmable gate arrays (FPGAs),complex programmable logic devices (CPLDs), and the like. Some or all ofthe system components and/or data structures may also be stored ascontents (e.g., as executable or other machine-readable softwareinstructions or structured data) on a computer-readable medium (e.g., ahard disk; memory; network; other computer-readable medium; or otherportable media article to be read by an appropriate drive or via anappropriate connection, such as a DVD or flash memory device) to enablethe computer-readable medium to execute or otherwise use or provide thecontents to perform at least some of the described techniques. Some orall of the components and/or data structures may be stored on tangible,non-transitory storage mediums. Some or all of the system components anddata structures may also be stored as data signals (e.g., by beingencoded as part of a carrier wave or included as part of an analog ordigital propagated signal) on a variety of computer-readabletransmission mediums, which are then transmitted, including acrosswireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). Such computer programproducts may also take other forms in other embodiments. Accordingly,embodiments of this disclosure may be practiced with other computersystem configurations.

All of the above U.S. Patents, U.S. Pat. Application Publications, U.S.Pat. Applications, foreign patents, foreign patent applications andnon-patent publications referred to in this specification and/or listedin the Application Data Sheet, including but not limited to U.S.Provisional Pat. Application No. 63/248,991, entitled “REMOTE LIVE SCENECONTROL SYSTEM, METHODS, AND TECHNIQUES,” filed Septermber 27, 2021, isincorporated herein by reference, in its entirety.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. For example, the methods and systems forperforming special effects discussed herein are applicable to otherspecial effects devices and for purposes other than those describedhere. Also, the methods and systems discussed herein are applicable todiffering protocols, communication media (optical, wireless, cable,etc.) and devices (such as wireless handsets, electronic organizers,personal digital assistants, portable email machines, game machines,pagers, navigation devices such as GPS receivers, etc.).

1. An apparatus for controlling a special effects device comprising: aprinted circuit board (PCB) comprising: a wireless networkmicrocontroller; a PCB header settable to configure a network addressassociated with the apparatus; a terminal connector configured toprovide power to the PCB; a data connector configured communicativelyconnect to one or more cables of a special effects device; and amicroprocessor including firmware; and a housing configured to house theprinted circuit board to allow the PCB to plug into a standardelectrical outlet to receive power and communicatively connect to thespecial effects device; wherein the firmware is configured to receive astream of data packets wirelessly using the wireless network controller,wherein each packet in the stream contains data structured according toa special effects device protocol, and wherein, for each packet, inresponse to special effects device protocol instructions contained inthe packet, the firmware generates one or more corresponding electrical,mechanical, and/or optical signals and transmits them to thecommunicatively connected special effects device to cause the specialeffects device to create a special effect.
 2. The apparatus of claim 1wherein the special effects device protocol is DMX.
 3. The apparatus ofclaim 1 wherein the special effects device is an LED device.
 4. Theapparatus of claim 3 wherein the special effects device is an LED stripand the special effect is a color change.
 5. The apparatus of claim 3wherein the special effects device is a Neopixel LED device and thecorresponding signals are formatted according to a corresponding driverspecification.
 6. The apparatus of claim 3 wherein the special effectcauses a change in color, color gradient, a timed color progression, ora pulsation of a single color or different colors.
 7. The apparatus ofclaim 1 wherein the special effects device is a fog machine or a smokemachine.
 8. The apparatus of claim 1 wherein the apparatus causes thespecial effect to occur in response to receiving a value sensed by asensor.
 9. The apparatus of claim 1 wherein the stream of data packetsis formulated according to an Art-Net protocol.
 10. The apparatus ofclaim 1 wherein the special effects device is a light wave device, amagnetic device, and/or an electronically controlled device.
 11. Asystem for remote control of special effects, comprising: a plurality ofspecial effects controllers, each communicatively connected to a specialeffects device, wherein each of the special effects controllers isconfigured to: communicate wirelessly over a network using a networkaddress associated with the respective special effects controller;receive one or more data packets with instructions according to aspecial effects protocol and translate each packet into a correspondingsignal compatible with the connected special effects device to cause aspecial effect to occur; and special effects enabled application codelogic, executing on a computer processor, and configured when executedto: discover and register two or more of the plurality of specialeffects controllers at the plurality of network addresses to create anad-hoc special effects synchronization zone; receive an indication of aspecial effect type and timing; and cause a stream of data packets to bedistributed wirelessly to one or more of the discovered and registeredplurality of special effect controllers according to the indicatedspecial effect type and timing, such that a synchronized special effectis caused to be performed on each of the special effects devices that iscommunicatively connected to a corresponding one or more of thediscovered and registered plurality of special effect controllers. 12.The system of claim 11, wherein at least one of the plurality of specialeffect controllers is located at a first network address and wherein atleast an other of the plurality of special effect controllers is locatedat a second network address, the first and second network addressesbeing at distinct locations from each other.
 13. The system of claim 11wherein the special effects protocol is a DMX protocol.
 14. The systemof claim 11 wherein the indicated special effect type and timing isassociated with a portion of an audio stream or audio data.
 15. Thesystem of claim 14 wherein the audio stream or audio data is beingdelivered live.
 16. The system of claim 15 wherein the audio stream oraudio data is prerecorded or pre-defined.
 17. The system of claim 14wherein the stream of data packets caused to be distributed wirelesslyto one or more of the discovered and registered plurality of specialeffect controllers is responsive to an analysis of audio data.
 18. Thesystem of claim 17 wherein the stream of data packets caused to bedistributed wirelessly to one or more of the discovered and registeredplurality of special effect controllers is able to be different uponperforming an analysis of a same audio data each time the analysis isperformed.
 19. The system of claim 11 wherein the special effectsenabled application code logic is configured when executed to receive anindication of a special effect type by receiving data from a sensorindicative of a condition for which the special effect type isappropriate.
 20. The system of claim 19 wherein the sensor detectsproximity.
 21. The system of claim 19 wherein the sensor detects atleast one of light wave, temperature, respiration, heart rate, and/orpulse.
 22. The system of claim 11 wherein the special effects enabledcode logic causes a synchronized special effect by causing a firstspecial effect to be performed on a first subset of the correspondingone or more of the special effect controllers and by causing a seconddifferent special effect to be performed on a second subset of the thecorresponding one or more of the special effect controllers.
 23. Amethod for controlling special effects, comprising: under control of aplurality of special effects controllers, each communicatively connectedto a special effects device, communicating wirelessly to receive one ormore data packets with instructions according to a special effectsprotocol and translating each packet into a corresponding signalcompatible with the connected special effects device to facilate causinga special effect to occur; and under control of special effects enabledapplication code logic, discovering and registering two or more of theplurality of special effects controllers at a plurality of networkaddresses to create an ad-hoc special effects synchronization zone;receiving an indication of a special effect type and timing; and causinga stream of data packets to be distributed wirelessly to one or more ofthe discovered and registered plurality of special effect controllersaccording to the indicated special effect type and timing, such that asynchronized special effect is caused to be performed on each of thespecial effects devices that is communicatively connected to acorresponding one or more of the discovered and registered plurality ofspecial effect controllers.